Multimodal Relevance Feedback for Interactive Image Retrieval

My research addresses the need for an efficient, effective and interactive access into large-scale image collections. In many cases the data of different modalities are interrelated , as for example photos and annotations in photo-sharing repositories , pictures and captions in news web-sites or x-ray scans and reports in medical databases, and I am investigating retrieval approaches that are capable of exploiting such interrelationships (ie. multimodal information retrieval). Most of the image retrieval approaches require an initial query before offering relevance feedback tools. The problem is that the user retrieval needs are often difficult to describe in terms of keywords and relevant images may be easily filtered out. Ferecatu and Geman proposed an innovative query-free approach. Starting from an heuristic sampling of the collection, this approach does not require any explicit query. It relies solely on an iterative relevance feedback mechanism. At each iteration, it displays a small set of images and the user is asked to show the image that best matches what he is searching for. After a few iterations, the displayed set starts to include images that satisfy the user. My main contribution so far is an extension of the state-of-the-art approach for annotated image collections. This extension integrates indexing information extracted from both image's visual content and their associated annotation keywords. Implemented as a web-application, the approach has been evaluated by 2 groups of 30 users each, and shown to be intuitive, easy to use and efficient. For a collection of 35000 images, the approach succeeds to retrieve images that satisfy the user in less than 5 iterations in 60% of the cases. The evaluation results obtained so far motivate further investigations. My research will remain focused on exploiting multimodal interrelationships for increasing the retrieval capabilities of this query-free approach. A particular goal will be to adapt the relevance feedback mechanism for large-scale applications and thus one step closer to commercial applications. The starting point will be to find ways to shrink the indexing information, which will reduce the storage capacity as well as the computational effort. Another goal will be to extend and evaluate the retrieval approach for other types of multimedia (eg. medical reports, songs, movies), hopefully in collaboration with other researchers.